Introduction

Magnetism in layered materials, held together by weak van-der-Waals interactions, is an active field of research spurred on by the discovery of magnetic ordering in monolayer sheets of ferromagnets and antiferromagnets1,2,3,4. At the frontier of this field, helimagnetic layered systems, where magnetic order has a fixed, left- or right-handed rotation sense, have been predicted to host complex spin textures5,6 and to serve as controllable multiferroics platforms, where magnetic order is readily tuned by electric fields or currents7,8,9,10. However, most layered van-der-Waals magnets are commensurate ferro-, antiferro-, or ferrimagnets3,4; the rare helimagnets provided to us by nature are often modulated along the stacking direction, with relatively simple spin arrangement in individual layers (Supplementary Table I).

In the quest for helimagnetism in layered structures with weak van-der-Waals bonds, we focus on rare earth tritellurides RTe3 (R: rare earth element). These materials form a highly active arena of research regarding the interplay of correlations and topological electronic states11,12,13,14,15. Their structure, which can be exfoliated down to the thickness of a few monolayers16,17, is composed of tellurium Te2 double-layers and covalently bonded RTe slabs, with characteristic square net motifs in both (Fig. 1a)18. Tellurium 5p electrons are localized in Te2 square net bilayers (ac plane), in which they form dispersive bands with elevated Fermi velocity16,19,20,21. This quasi two-dimensional electronic structure amplifies correlation phenomena, such as the formation of charge density wave (CDW) order22 and superconductivity23,24, while also hosting protected band degeneracies16,25. In view of intense research efforts on RTe3, it is remarkable that their magnetism has never been discussed in detail; in particular, no full refinement of magnetic structures is available26,27,28,29,30,31.

Fig. 1: Conical helimagnetism in the layered square-lattice antiferromagnet DyTe3.
figure 1

a Crystallographic unit cell with covalently bonded DyTe bilayers and metallic Te bilayers. b Magnetic exchange interactions in a single DyTe double-square net bilayer. c Zigzag chain illustration of double-square net structure in DyTe3. The interactions J1, J2 connect nearest and next-nearest neighbors in the zigzag chain model, respectively; but the inter-atomic distance is shorter for J2, and its antiferromagnetic coupling strength is dominant. Conical, noncoplanar helimagnetism is resolved in the zero-field ground state by neutron scattering. The cone direction, parallel to the a-axis, alternates both between pairs of magnetic sites in a zigzag chain, and between stacked zigzag chains. This texture causes polarization along the b-axis, i.e., perpendicular to the square net bilayers (yellow arrow). The full magnetic unit cell extends two times further along the chain direction (Supplementary Fig. 1). d Weak anisotropy of the magnetic susceptibility χ in DyTe3. The softest direction is H∥c, consistent with the modulation direction of the magnetic order in (c). The inset shows Curie–Weiss temperatures and temperatures Tχ of maximal χ, measured in a small magnetic field along three crystallographic directions. Error bars correspond to statistical uncertainties of Curie–Weiss fitting and temperature sampling width, respectively.

Full size image

Here, we report on helimagnetic, cone-type orders of DyTe3 using polarized elastic neutron scattering. We reveal the magnetic texture in real space, probe its evolution with temperature and magnetic field, and discuss its relationship to CDW formation. In DyTe3, dysprosium moments are arranged in square net bilayers, where each ion has neighbors within its own layer, and within the respective other layer (Fig. 1b). As all magnetic orders of DyTe3 observed here are uniform along the crystallographic a-axis, it is reasonable to understand each square net bilayer as an effective zigzag chain of magnetic rare earth ions and to define magnetic interactions J1 and J2 in terms of nearest- and next-nearest neighbors on the zigzag chain, respectively. On such chains, our experiment shows that pairs of ions have cones pointing along the same direction, followed by a flip of the cone axis (Fig. 1c, which illustrates half a magnetic unit cell). The coupling between two DyTe bilayers, i.e., between two zigzag chains, is antiferromagnetic. Despite this complex cone arrangement, the magnetic structure defines a fixed sense of rotation, or helicity. Our theoretical spin model shows that CDW order in rare earth tellurides causes local symmetry breaking and drives lattice-incommensurate magnetism when combined with antiferromagnetic interactions or with a net magnetization. We further discuss magneto-crystalline anisotropy in this layered structure, with an unconventional combination of metallic and covalent bonds. Helimagnetism of Dy rare earth moments with 4f 9 magnetic shell emerges despite the naive expectation of easy-axis or easy-plane anisotropy for 2S+1LJ = 6H15/2, with large orbital angular momentum L = 5.

Results

Magnetic properties of DyTe3

Some essential magnetic properties of DyTe3 are apparent already from the magnetic susceptibility χ in Fig. 1d. In the high temperature regime, anisotropy in the Curie–Weiss law indicates easy-plane behavior of magnetic moments, favouring the ac plane with uniaxial anisotropy constant K1 = 22(1) kJ m−3, see “Methods”. At low temperatures, the strongest enhancement of χ occurs when the magnetic field H is along the c-axis, i.e., parallel to the zigzag direction defined in Fig. 1c. We deduce that the magnetic moments are aligned, predominantly, along the a- and b-axes. All susceptibility curves show maxima around Tχ = 4.5 K, quite far above the onset of three-dimensional, long-range magnetic order, as shown in the following.

We characterize the phase transition in DyTe3 using thermodynamic and transport probes in Fig. 2a, b. The specific heat C(T) shows a two-peak anomaly, describing the transitions from the paramagnetic regime to Phase II at TN2 = 3.85 K and to Phase I at TN1 = 3.6 K. Below TN1, the resistivities in the ac basal plane drop abruptly, suggesting a clear correlation between the behavior of freely moving conduction electrons and the magnetic structure. The presence of a partial charge gap in the electronic structure, related to magnetic ordering, is inferred from an increase of the ratio of resistivities ρa and ρc. Simultaneously, as discussed in the following, strong neutron scattering intensity appears below TN2 at two independent positions in reciprocal space, c.f. Fig. 2c. The magnetic scattering intensity rises abruptly upon cooling below TN2.

Fig. 2: Bulk characterization and zero-field magnetic structure of DyTe3 from elastic neutron scattering.
figure 2

a Specific heat showing two transitions at TN1 and TN2, with reference data from the nonmagnetic analog LaTe332. b Below TN1, where resistivity in the ac basal plane drops sharply, significant anisotropy ρc/ρa indicates (partial) gapping of electronic states along the c-axis. c Temperature dependence of magnetic scattering intensity for antiferromagnetic qAFM and cycloidal qcyc reflections, with Miller indices (HKL) = (0, 1, 0.5) and (0, 1, 0.207), respectively. Gray and green background shading mark Phases I and II, respectively. Error bars correspond to statistical uncertainties of Gaussian fits to the magnetic reflections. d Experimental geometry for polarized neutron scattering from layered DyTe3 (gray plane, square net of Dy indicated). The scattering plane (blue) is spanned by wavevectors ki, kf of incoming and outgoing neutron beams, respectively. Separating spin flip (SF, red) and non-spin flip (NSF, blue) scattering intensities at the detector, we identify conical magnetic order (top). e Linescan in momentum space, with highlights for three types of reflections including a weak higher harmonic corresponding to 3 × qcyc. f Simulation of magnetic intensity from magnetic structure model, corresponding to the linescan in (e), see “Methods”.

Full size image

To obtain this neutron data, a single-domain crystal of DyTe3 is mounted on an aluminum holder and is pre-aligned by means of Laue X-ray diffraction. More quantitatively, we determine the crystallographic directions in DyTe3 using the crystallographic extinction rule (Supplementary Fig. 11). Figure 2d describes the geometry of our neutron scattering experiment. The scattering plane that includes the incoming and outgoing neutron beams ki and kf, is spanned by the b- and c-axes. Hence, reflections of the type Q = kf − ki with Miller indices (0KL) can be detected, as in Fig. 2e, where a line scan along (01L) provides sharp magnetic intensity. Several types of magnetic peaks Q = G + q, with G a reciprocal lattice vector, are observed: primarily, a commensurate (C) reflection qAFM = (0, b*, qAFM), qAFM = 0.5c* and an incommensurate (IC) reflection qcyc = (0, b*, qcyc), qcyc = 0.207 c*, where b* = 2π/b and c* = 2π/c are reciprocal lattice constants, see “Methods”. There is also a higher harmonic (3Q) reflection, corresponding to three times the length of qcyc, which describes an anharmonic distortion of the texture. As a main result of this work, we ascribe qcyc to a cycloidal structure in the magnetic ground state of DyTe3, that results from a coupling qAFM ± qCDW between qAFM and a CDW modulation qCDW at (0, 0, qCDW), qCDW = 0.29 c* in the rare earth tritelluride family.

Ground state magnetic structure model

We reveal the helimagnetic structure in the ground state of DyTe3 using polarized neutron scattering. As shown in Fig. 2d, the incident neutron spins are polarized perpendicular to the scattering plane. We employ a magnetized single-crystal analyzer to select the energy and spin state of the scattered neutrons, see “Methods”. The scattering processes in which the neutron spins are reversed (remain unchanged) is referred to as spin-flip, SF (non-spin-flip, NSF). For SF scattering, it is required that the magnetic moments m have a component perpendicular to the spin of the incoming neutron. This means that SF and NSF scattering detect components of m within (mb, mc) and perpendicular to (ma) the scattering plane, respectively.

Polarization analysis of the magnetic reflections shows that qcyc and qAFM relate to different vector components of the ordered magnetic moment (Fig. 3a, b, e, f). We find no hint of SF scattering at qAFM, demonstrating collinear antiferromagnetism with magnetic moments exclusively along the a-direction. The incommensurate part qcyc, in contrast, has no NSF intensity and roughly equal SF signals at various positions in reciprocal space (Fig. 3e, f and insets). As neutron scattering detects the part of m that is orthogonal to Q, comparison of magnetic reflections situated at nearly orthogonal directions in momentum space suggests mb and mc components are both finite in the ground state.

Fig. 3: Two spin components of the cone-type order in the ground state of DyTe3 (Phase I).
figure 3

In the geometry of Fig. 2d, non-spin-flip (NSF) and spin-flip (SF) neutron scattering measure the magnetic moment along the crystallographic a-axis and in the bc plane, respectively. a, b Polarization analysis of antiferromagnetic component qAFM: absence of spin flip (SF) intensity indicates absence of mb and mc, while non-spin flip (NSF) intensity reveals dominant ma. c, d Two views of the antiferromagnetic collinear component derived from this data, confirmed by full refinement of a number of magnetic reflections in Supplementary Fig. 8. Only Dy atoms are shown. e, f Polarization analysis for the incommensurate cycloidal reflection qcyc. g Derived magnetic structure model for qcyc, where ma (NSF) vanishes while mc and mb (SF) both appear at comparable magnitudes in Phase I. The index j labels four dysprosium atoms in four layers within the chemical unit cell (red box), where the cycloids at j = 2, 4 are phase-shifted with respect to cycloids at j = 1, 3. The numerical value of δ is determined in Supplementary Section VI. As for a, b, e, f we have subtracted background signals, and corrected the effect of imperfect beam polarization. The error bars correspond to Poisson counting errors of the integrated neutron scattering intensity, given in counts per second (cps). Each inset depicts the orientation of the measured Q-vector within the (0KL) reciprocal space.

Full size image

We determine the quantitative relationship between magnetic moments within a DyTe bilayer (within an effective zigzag chain), by comparing the observed and calculated magnetic structure factors under the constraints imposed by polarized neutron scattering, c.f. Supplementary Fig. 8. The analysis for qAFM demonstrates up–up–down–down type ordering along the zigzag chain, visualized from two perspectives in Fig. 3c, d. At qcyc, the refinement yields a cycloid with a phase delay δ between the upper and lower sheets in a zigzag chain, see Fig. 3g. In effect, pairs of nearly parallel magnetic moments are followed by a significant rotation of the moment direction. The coupling between zigzag chains is antiferromagnetic, as imposed by the Miller index K = 1 (qb = b*) in both qcyc and qAFM. Combining the three components ma, mb, and mc, we realize the noncoplanar, helimagnetic cone texture of Fig. 1c that is, to our knowledge, unique in both insulators and metals. In Supplementary Sections III A, IV A, we discuss the presence of magnetic domains in the sample and how the occurrence of higher harmonic reflections further supports our magnetic structure model.

Charge density wave and magnetic order

We now argue that cone-type magnetism in DyTe3 is realized through (i) coupling of the magnetic texture to CDW order and (ii) unconventionally weak single-ion anisotropy. We turn first to (i), that is the role of the CDW in stabilizing noncoplanar helimagnetism in DyTe3. We use a 1D chain model to reproduce key features of the modulated magnetic order, neglecting the material’s three-dimensionality, see “Methods”.

In DyTe3, the local environment and bond characteristics of dysprosium ions in a DyTe square net bilayer (in a zigzag chain) are spatially modulated by the CDW in the adjacent Te2 sheets, c.f. Fig. 4b11,12,13,14,18,22,23,24,31,32,33. The simplest model approach is to decouple the zigzag chain, with two atoms per unit cell, into two one-dimensional chains, with one atom per unit cell. This allows for a two-parameter model, built from Ising-like exchange interactions together with a spatially modulated on-site coupling,

$${{{{{{{\mathcal{H}}}}}}}}={\sum}_{n}\left[{J}_{2}^{{{{{{{{\rm{AFM}}}}}}}}}{S}_{n}^{a}{S}_{n+1}^{a}-{E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ab}\cos \left({q}_{{{{{{{{\rm{CDW}}}}}}}}}{z}_{n}\right){S}_{n}^{a}{S}_{n}^{b}-{E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ac}\sin \left({q}_{{{{{{{{\rm{CDW}}}}}}}}}{z}_{n}\right){S}_{n}^{a}{S}_{n}^{c}\right]$$
(1)

where n counts magnetic sites, e.g., on the upper half of the zigzag chain. The zn are spatial positions along the zigzag chain (c-axis). All the coupling constants – \({J}_{2}^{{{{{{{{\rm{AFM}}}}}}}}}\), \({E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ab}\), and \({E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ac}\) – are positive. The \({E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ab}\) and \({E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ac}\) terms are allowed by global and local mirror symmetry breaking due to the CDW, respectively. We may also introduce a Zeemann term to explain the behavior in a magnetic field and further inter-chain coupling to connect the two chains (Supplementary Section I).

Fig. 4: Modeling helimagnetism in DyTe3.
figure 4

a Magnetic susceptibility χ for H∥[101], overlaid with phase boundaries from specific heat (white open circles) and magnetization (black circles): We label phases I (conical ground state), II, III, IV, and V. b CDW on tellurium layers18, where orange (gray) spheres are distorted (undistorted) ionic positions. Coupling of Dy (violet) and CDW drives lattice-incommensurate magnetic order through spatially modulated distortion of the local environment of Dy-ions. c Squared moment amplitudes \({I}_{{{{{{{{\rm{AFM}}}}}}}}}={\left\vert {S}^{a}(q=0.5{c}^{*})\right\vert }^{2}\) (blue) and \({I}_{{{{{{{{\rm{cyc}}}}}}}}}={\left\vert {S}^{b}(q=0.5{c}^{*}\pm {q}_{{{{{{{{\rm{CDW}}}}}}}}})\right\vert }^{2}+{\left\vert {S}^{c}(q=0.5{c}^{*}\pm {q}_{{{{{{{{\rm{CDW}}}}}}}}})\right\vert }^{2}\) (red) as functions of \({E}_{{{{{{{{\rm{CDW}}}}}}}}}/{J}_{2}^{{{{{{{{\rm{AFM}}}}}}}}}\), where \({E}_{{{{{{{{\rm{CDW}}}}}}}}}={E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ab}={E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ac}\), from model calculations according to Eq. (1); five and ten percent threshold for the incommensurate part indicated by black and gray dashed lines, see “Methods”. d Local environment of Dy in DyTe3, with covalent (metallic) bonds to Te-B (Te-A) depicted by solid (dashed) lines, respectively. e Charge density (CD) of Dy 4f 9 shell under the influence of crystal electric fields (CEF) from the surrounding ions, with exaggerated non-spherical part. If metallic and covalent bonds cause CEF of roughly equal strength (if metallic bonds are screened), oblate \(\left\vert\, {J}_{b}=\pm \!15/2\right. \rangle\) (prolate \(\left\vert \, {J}_{b}=\pm \!1/2\right. \rangle\)) is the lowest energy CEF doublet. This favors out-of-plane (in-plane) magnetization, respectively. Color on CD isosurfaces indicates amplitude \(\left\vert {{{{{{{\bf{E}}}}}}}}\right\vert\) of the local CEF. f, g Anisotropy constants K1 and K2 calculated for 4f 9 multiplet in DyTe3, see “Methods”. The abscissa describes the relative weight of CEF charges c on Te-A and Te-B sites. Green lines bound an intermediate regime of weak K1, where spin tilting and conical order are allowed.

Full size image

This 1D model naturally creates different modulation period for the a and bc spin components and robustly reproduces two types of magnetic reflections, qAFM/c* = 0.5 and qcyc/c* = 0.5 − 0.293 = 0.207. In good consistency with the experiment, Fig. 4c shows that Icyc on the order of 10% of IAFM can be induced within this model. Note that, in addition to the magnetic texture in Fig. 1c, alternative (out-of-phase) locking between cycloid and antiferromagnetic component is also possible (Supplementary Fig. 1). Based on scattering techniques alone, it is not possible to decide the phase-shift between CDW and the spin cycloid, and between the antiferromagnetic and incommensurate components of the magnetic order.

Weak magneto-crystalline anisotropy

While strongly anisotropic magnetism is naively expected for dysprosium’s 6H15/2 shell, we here report a conical state with comparable ma, mb, mc in DyTe3. Consider the local environment of a single Dy in Fig. 4d: Te-B ions form covalent bonds with the central Dy, while the point charges of Te-A are effectively screened by itinerant electrons in the conducting tellurium slab. We model the sequence of crystal electric field (CEF) states for the 4f 9 shell of dysprosium as a function of the effective crystal field charge c situated on Te-A and Te-B ions (Supplementary Fig. 15). Figure 4e illustrates two limiting cases: when Te-A and Te-B contribute equally to the CEF, the 4f 9 charge cloud is compressed along the b-direction, with \(\left\vert\, {J}_{b}=\pm \!15/2\right. \rangle\) dominating the ground state wavefunction, and with effective out-of-plane magnetic anisotropy. Likewise, zero contribution of Te-A, i.e., highly efficient metallic screening of CEFs, favors the prolate orbital \(\left\vert\, {J}_{b}=\pm \!1/2\right. \rangle\) with easy-axis anisotropy.

Adding exchange interactions Eex as an effective magnetic field, the CEF Hamiltonian of a point charge model is diagonalized for the orthorhombic environment of Dy, see “Methods”. The resulting free energy density described by two parameters \({K}_{1}{\cos }^{2}(\theta )+{K}_{2}{\sin }^{2}(\theta ){\cos }^{2}(\phi )\), where θ, ϕ are spherical coordinates with respect to the b and c crystal axes, respectively. Figure 4f, g testifies to a transition from easy-axis to easy-plane anisotropy through a sign change of K1 at intermediate charge ratio (pink line). Two green lines bound the regime where easy-axis (easy-plane) anisotropy is not strong enough to prevent tilting of m along directions intermediate between b-axis and the ac-plane. Constraining Eex in agreement with TN2 and requiring easy-plane anisotropy K1 > 0, we identify the black box in Fig. 4f, g to capture a parameter range well consistent with experiment. Here, the model yields K2 > 0, meaning ma is preferred over mc.

Magnetic phase diagram and small-angle neutron scattering

We are ready, now, to consider the evolution of magnetic order in DyTe3 as a function of temperature and magnetic field. Figure 4a shows a contour map of the magnetic susceptibility χ, see “Methods”, where the external magnetic field is applied along the in-plane direction [101], i.e., H∥(a + c). Heating the sample above TN1 = 3.6 K in zero field, we observe a peak splitting of qAFM, and a concomitant shift in qcyc that indicates the sustained coupling of the two ordering vectors, via the CDW, at elevated temperatures (Supplementary Fig. 13). The sharp enhancement of χc in Fig. 1a further suggests that ma, mb survive to higher temperature than mc, consistent with a putative incommensurate, fan-like order in Phase II, which warrants further study.

To explore the high-field regime, confirm the coupling between qAFM and qcyc in Phase I, and investigate the correlation of CDW and magnetic order, we carried out small-angle neutron scattering (SANS) experiments in a magnetic field. Figure 5a describes the geometry of our SANS experiment and Fig. 5b shows the obtained zero field (0, K, L) map, while Fig. 5c reduces the map into principal line cuts. The non-coplanar, helimagnetic cone texture with coupled qAFM and qcyc is stable up to μ0H = 0.5 T for H∥c, c.f. Fig. 5f. In fact, these data further support the existence of coupled commensurate and incommensurate order parameters in Phase I, and help to exclude a domain separation scenario (Supplementary Fig. 14). A pair of phase transitions to Phases IV and V is visualized in Fig. 5f and Supplementary Fig. 14. In Phase V, c.f. Fig. 5d, e, which is realized when the external magnetic field exceeds μ0H = 0.7 T, strong magnetic reflections appear at momentum transfer Q = (Ha*, Kb*, qCDW) with K = even, demonstrating direct coupling between incommensurate magnetism and the CDW in absence of the antiferromagnetic order parameter. Although we cannot provide a magnetic structure model based on the available data, the comparison of intensities at Miller indices K = 0, 2 suggests the presence of both ma and mb spin components (inset of Fig. 5e), consistent with longitudinal conical magnetic order.

Fig. 5: Coupling of magnetic order and charge density wave in DyTe3, tracked by small-angle neutron scattering (SANS) in a magnetic field along the c-axis at T = 2 K.
figure 5

a Experimental geometry of the SANS experiment. Orange plate, blue rectangle, kin, kout, and Q are the sample, the area detector – shifted from the center axis to catch reflections with larger \(\left\vert {{{{{{{\bf{Q}}}}}}}}\right\vert\) – incoming/outgoing neutron beam wavevector, and the momentum transfer of elastic scattering, respectively. A wedge of three-dimensional Q space is covered by rotating the sample in steps (angle ω, see “Methods”). b–e Scattering intensity in (0KL) cuts of momentum space (upper) and linecuts along (00L) (lower), for intensity integrated along the K-direction within the orange and red dashed boxes. The magnetic field induces a transition to ferroic stacking, K = 0. f Integrated intensity of various reflections as a function of magnetic field, covering three magnetic phases. Error bars correspond to statistical uncertainties of Gaussian fits to the extracted linecuts, c.f. c, e. Inset of e: experimental ratio γ = I(0, −2, −0.293)/I(0, 0, −0.293) in Phase V, compared to simulations for fan-like orders (spin plane indicated, green and gray dashed lines) and a helimagnetic (longitudinal-cone, red dashed line) order. The experiments are consistent with a slightly distorted, conical state of the same period as the underlying charge order. Error bars are derived by error propagation from the statistical uncertainties in (f).

Full size image

Discussion

As compared to transition metal dichalcogenides, where the magnetic ion is buried inside a rather symmetric block layer34,35, RTe3 harbors more complex structural features, with magnetic ions at the boundary between metallic and covalently bonded blocks. This mixed covalent/metallic environment for the magnetic ion is key to realizing the present scenario: it facilitates coupling between magnetic ions and a CDW on the tellurium square net, and – at the same time – generates unconventional magneto-crystalline anisotropy. In fact, the present charge-transfer phenomenology in the point charge model is partially inspired by work on thin films of magnetic metals on insulating substrates36, on electric field control of magneto-crystalline anisotropy37, and on the behavior of magnetic materials when charge transfer is induced by oxidation at the surface38.

Symmetry breaking with cycloid/spiral magnetic order of fixed helicity is rather widely observed in zigzag chain magnets (Supplementary Section IV A), but the present combination of antiferromagnetic and cycloidal components is unique. For example, Mn2GeO4 has cones arrayed on one-dimensional chains, with uniform cone direction along the chain39. In another metallic system, EuIn2As2, jumps in the rotation of a helimagnetic texture have recently been identified, with a short magnetic period40. In contrast, the rotation of moments in DyTe3 proceeds with nearly parallel pairs of spins. We expect helimagnetic orders of the type observed here to be common in layered materials, and especially in rare earth tellurides and selenides that await exploration by neutron diffraction. Here, rich magnetic phase diagrams have been generally observed41,42 and could be amenable to modeling by CDW-induced terms as in Eq. (1).

DyTe3’s complex magnetic order, its relationship to a strain-controllable CDW33, and its (likely) rich excitation spectrum certainly warrant further research. For example, the CDW’s gapped collective excitation, termed Higgs mode, shows an axial character in RTe3 as observed via Raman scattering experiments15, and should thus have a spatially modulated (electric or magnetic) moment. The evolution of this unconventional CDW below TN1 may provide insights on both the origin of magnetic order and the nature of the CDW in DyTe3. Furthermore, the lowest energy, Goldstone mode of a typical helimagnet corresponds to a spatial shift of the magnetic texture, termed phason excitation43. In DyTe3, the magnetic and CDW phasons44 are expected to be closely intertwined, as evident from the robust qcyc(T) in Phase I, and its jump – by the same amount as qAFM – in Phase II (Supplementary Fig. 13). Such locking between low-energy modes may have implications for dynamic responses, further enriching the spectrum of elementary excitations in RTe3.

An important open question is the stability of helimagnetism in few-layer devices of DyTe3. As a fundamental building block of the structure, we consider a DyTe slab sandwiched by Te square nets – that is half a unit cell in Fig. 1a. Being screened from top and bottom by tellurium layers, we expect no qualitative change of the local crystal field environment of Dy in the few-layer limit. However, the absence of an inversion center for odd numbers of layers, and its presence for even numbers of layers, may have a profound effect on magnetic ordering and the presence or absence of (right- or left-handed) helicity domains in the sample, considering the presence or absence of Dzyaloshinskii–Moriya interactions45.

Most appealingly, DyTe3 is a potential platform for spin-moiré engineering in solids, where complex magnetic textures can be designed by combining and twisting two or more helimagnetic sheets. Here, a plethora of noncoplanar spin textures can be engineered at will6,46, while highly conducting tellurium square net channels may serve as a test bed for emergent electromagnetism in a tightly controlled setting47,48.

Methods

Sample preparation and characterization

Single crystals are grown from tellurium self-flux following the recipe in ref. 42: we set elemental Dy and Te at a ratio of 1:21.65 in an alumina crucible, which in turn is sealed in a quartz tube in high vacuum. The raw materials are heated to 450 °C for 36 h and then to 780 °C in 96 h, where the melt remained for 48 h, followed by cooling to 450 °C at a rate of 1.375 °C/h. The final product is centrifuged after renewed heating to 500 °C, so that plate-shaped single crystals of typical dimensions 5 × 5 × 1 mm3 are obtained. The face of each plate is perpendicular to the b-axis of DyTe3’s orthorhombic unit cell, and facet edges tend to be parallel to either a or c. The existence of impurity phases above 1% volume fraction is ruled out by single-crystal X-ray diffraction on cleaved surfaces in a Rigaku SmartLab X-ray powder diffractometer. The experiment yields lattice constants of a = 4.27(2) Å, b = 25.433(1) Å, and c = 4.27(2) Å at room temperature, in good agreement with previous work18. We found it challenging to obtain high-quality powder X-ray data from crushed single crystals, which include traces of Te flux on their surface and form thin flakes, even when thoroughly ground in a mortar. We also verified the stoichiometric chemical composition of our crystals by energy-dispersive X-ray spectroscopy. Cleaved single crystals have a reddish-brown surface; but even in vacuum, the color of the surface changes to silver-metallic, and then to black, after 2 weeks or so. A red hue can be recovered by renewed surface cleaving.

Magnetization measurements and crystal alignment

We use a commercial magnetometer with T = 2 K base temperature and a maximum magnetic field of 7 T (MPMS, Quantum Design, USA). The measurement is carried out using a rectangular-shaped single crystal of mass m = 1.22 mg, with carefully aligned edges along the a and c crystal axes. By means of a single-crystal diffractometer (Malvern Panalytical Empyrean, Netherlands), we confirm the extinction rule h + k = even in space group Cmcm. It is difficult to distinguish a- and c-axes in this orthorhombic, yet nearly tetragonal structure by eye or with the help of a Laue diffractogram. Temperature dependent susceptibility χ(T) is measured in a DC magnetometer with an applied field of 1000 Oe; there is no observable difference between field-cooled and zero-field cooled magnetization traces. A demagnetization correction is carried out according to the standard expression Hint = Hext − NM, where Hext, M, and N are the externally applied magnetic field, the bulk magnetization, and the dimensionless demagnetization factor. The latter is calculated by approximating the crystal as an oblate ellipsoid49. We obtain the demagnetization-corrected susceptibility as χint = M / Hint.  For the H − T phase diagram in Fig. 4a, the [101] direction is aligned within ±3° and bulk magnetization is measured in discrete field steps, for selected temperatures. Figure 4a shows data for decreasing magnetic field ∂H/∂t < 0. Note that hysteresis occurs at all phase transitions shown in Fig. 4a, indicating their first-order nature. The magnetic anisotropy energy is expressed as \(E/{V}_{{{{{{{{\rm{uc}}}}}}}}}={K}_{0}+{K}_{1}{\cos }^{2}(\theta )+{{{{{{{\mathcal{O}}}}}}}}[{\sin }^{4}(\theta )]\), where θ is the angle between M and the b-axis. Utilizing the free energy expression \(F={F}_{0}+{\sum }_{\alpha }{a}_{i}(T){M}_{\alpha }^{2}+{{{{{{{\mathcal{O}}}}}}}}({{{{{{{{\bf{M}}}}}}}}}^{4})\) and the Curie–Weiss law \({\chi }_{\alpha }=C/(T-{\Theta }_{{{{{{{{\rm{CW}}}}}}}}}^{\alpha })\), where \({\Theta }_{{{{{{{{\rm{CW}}}}}}}}}^{\alpha }\) is the Curie–Weiss temperature along the α ∈ (a, b, c) direction and C = 2.077(1) K is the Curie constant of DyTe3, we obtain \({K}_{1}=({\mu }_{0}/2){\left(M/{V}_{{{{{{{{\rm{uc}}}}}}}}}\right)}^{2}(\Delta {\Theta }_{{{{{{{{\rm{CW}}}}}}}}}/C)=22(1)\,{{{{{{{\rm{kJ}}}}}}}}\,{{{{{{{{\rm{m}}}}}}}}}^{-3}\), where \(\Delta {\Theta }_{{{{{{{{\rm{CW}}}}}}}}}={\Theta }_{{{{{{{{\rm{CW}}}}}}}}}^{ac}-{\Theta }_{{{{{{{{\rm{CW}}}}}}}}}^{b}=-2.0(1)\,{{{{{{{\rm{K}}}}}}}}\) is the difference between the Curie–Weiss temperatures in the ac-plane and along the b-axis (c.f. Fig. 1d, inset). Specific heat was recorded using a relaxation technique in a Quantum Design PPMS cryostat, in zero magnetic field. For specific heat anomalies in applied magnetic field, we employed the AC calorimetry technique in a custom-built setup. Anisotropy of the resistivity, as in Fig. 2b, was recorded on exfoliated flakes of thickness ~100 μm using the Montgomery technique. Electric contacts are made with Ag paste (Dupont) and deteriorate with time. To maintain excellent contact resistance ~1 Ω, it is crucial to immediately cool the contacted crystal in vacuum, after depositing the silver paste. The sample and contact quality is robust at low temperatures for at least 2 weeks.

Elastic neutron scattering

We performed unpolarized and polarized neutron scattering experiments using the POlarized Neutron Triple-Axis spectrometer (PONTA) installed at the 5G beam hole of the Japan Research Reactor 3. Two single crystals of DyTe3 (Samples A and B) are cut into rectangular shapes with dimension 3.6 × 2.7 × 0.9 mm and 1.7 × 1.9 × 0.8 mm, respectively. For both samples, the widest surface is normal to the b-axis, and the sides are parallel to the a- or c-axis. Each sample is set in an aluminum cell, which is sealed with 4He gas for thermal exchange. We employed a 4He closed-cycle refrigerator with base temperature of 2.2 K, and measured intensities on the (0, K, L) horizontal scattering plane. Using a PG (002) monochromator, the energy of the incoming neutron beam is set to Ei = 14.7 meV (30.5 meV) for upolarized measurements of Sample A (Sample B). For the unpolarized measurements, the spectrometer is operated in two-axis mode with horizontal beam collimation of open-\(8{0}^{{\prime} }\)-\(8{0}^{{\prime} }\). In both unpolarized and polarized experiments, sapphire and pyrolytic graphite (PG) filters are installed between the monochromator and the sample, to suppress higher-order reflections from the monochromator to <0.5%. The observed integrated intensities are converted to structure factors after applying the Lorentz factor and absorption corrections.

For Sample B, we measured nuclear and magnetic Bragg reflections at 2.2 K by θ − 2θ scans. For the scattering profiles showing a well-defined Gaussian-shape peak, we estimated the background from both ends of the profile. For the magnetic reflections located near the powder diffraction lines of the Al sample holder, we carried out background scans at 10 K, and subtracted the intensities from those measured at 2.2 K. We also measured the background data at 10 K for relatively weak commensurate magnetic reflections in the Q-range of \(\left\vert Q\, \right\vert \, > \, 4.0\) Å−1, to check for possible λ/2 contamination from the nuclear reflections. As for the absorption correction, we calculated the scattering path length l inside the sample, based on the dimensions of Sample B and on the incident and scattered directions of the neutrons. The neutron transmission is given by \(\exp (-\mu l)\), where μ is the linear absorption coefficient. Taking into account the incident energy, the absorption and the incoherent scattering cross-sections of DyTe3, μ is calculated to be 8.392 cm−1.

The diffraction profiles and integrated intensities shown in Figs. 2 and 3 were measured using Sample A. Contrary to integrated intensities in the case of refinement, the temperature dependences in Fig. 2c and Supplementary Fig. 13 are obtained from L-scans of magnetic scattering. The calculation of the scattering intensity in Fig. 2f, which includes the third harmonic reflection, takes into account instrumental resolution broadening (Supplementary Fig. 10), anharmonicity of the cycloidal magnetic structure component, and the presence of two magnetic domains (Supplementary Section IV).

Sample A is also used for polarized neutron scattering, in which the spectrometer is operated in the triple-axis mode with horizontal beam collimation of open-\(8{0}^{{\prime} }\)-\(8{0}^{{\prime} }\)-open. A polarized neutron beam with Ei = 13.7 meV is obtained by a Heusler (111) crystal monochromator. The spin direction of incident neutrons is set to be perpendicular to the scattering plane. We thus applied weak vertical magnetic fields of ~5 mT throughout the beam path by guide magnets and a Helmholtz coil. We used a Mezei-type π spin flipper placed between monochromator and sample, and employed a Heusler (111) crystal analyzer to select the energy and spin states of scattered neutrons, separating SF and NSF intensities. The spin polarization of the incident neutron beam (P0) is 0.823, as measured using the (002) nuclear Bragg reflection of the sample.

Small-angle neutron scattering in magnetic field

SANS measurements were performed using the SANS-I instrument at Paul Scherrer Institute, Switzerland. A bulk single crystal of DyTe3 (Sample E, m = 73.4 mg) was carefully aligned (c.f. crystal alignment in “Methods”) and installed into a 1.8 T horizontal-field cryomagnet so that the a-axis is vertical, and the incident neutron beam is in the bc-plane. The magnetic field is applied parallel to the crystal c-axis, as shown in Fig. 5a. The incident neutron beam with λ = 3.1 Å wavelength (15% Δλ/λ) is collimated over a distance of 4.5 m before the sample, and the scattered neutrons are detected by a 1 m2 two-dimensional multidetector (pixel size 7.5 mm × 7.5 mm) placed 1.7 m behind the sample. To cover a broader q-space up to qAFM along the (01L) direction, the detector was also translated 0.45 m in the horizontal plane. For all SANS data, background signals from the sample and the instrument are subtracted using the data of the nonmagnetic state at T = 10 K and μ0H = 0 T. The field-dependent SANS measurements are performed during a field-increasing process, after an initial zero-field cool to the base temperature of 2 K. For each measurement, rocking scans were performed, i.e., the cryomagnet is rotated together with the sample around the vertical crystal a-axes (rocking angle ω) in a range from −102° to 55° and steps of 2° (−38° ≤ ω ≤ 28°) and 1° (else). Here ω = 0° is carefully aligned and corresponds to the configuration where the beam kin is parallel to the crystallographic b-axis. The SANS maps shown in this paper are obtained by performing a 2D cut of the volume of reciprocal space measured through cumulative detector measurements taken at each angle of the rocking scan. For the SANS maps shown in Fig. 5, the integration width along the out-of-plane (H00) direction is ±0.15 reciprocal lattice units. The line cuts along (0KfixL) shown in Fig. 5c, e are extracted by integrating over a region of ±0.2 reciprocal lattice units in the (0K0) direction. Peak positions and integrated intensities are calculated using those linecuts and a multi-peak fitting.

Crystal electric field calculations

We use the software package PyCrystalField50 for the calculation of CEF energies via the point charge model in the limit of strong spin–orbit interactions. The calculation is based on published fractional coordinates of Dy and Te ions within the crystallographic unit cell18,22, with a ~0.15% tensile strain along the a-axis, lifting tetragonal symmetry and yielding finite K2. In Supplementary Fig. 15, we vary the effective CEF originating from Te-A (on the Te2 slab) by changing its point charge, while keeping the total charge in the environment of Dy unchanged. An unperturbed, diagonal Hamiltonian matrix is constructed from the energies in Supplementary Fig. 15, and the operator of total angular momentum J = L + S is also expressed in the basis of these CEF eigenstates. Adding an effective exchange field EexJα (α = a, b, c are vector components), the total Hamiltonian is diagonalized and the expectation value of Ja, Jb, Jc is evaluated in the respective ground state. The anisotropy constants are approximated, as

$${K}_{1}\propto \frac{1}{{ \langle\, {J}_{b} \rangle }_{b}^{2}}-\frac{1}{{ \langle\, {J}_{a} \rangle }_{a}^{2}}$$
(2)
$${K}_{2}\propto \frac{1}{{ \langle\, {J}_{c} \rangle }_{c}^{2}}-\frac{1}{{ \langle \,{J}_{a} \rangle }_{a}^{2}}$$
(3)

so that K1 < 0 for easy-axis anisotropy along the b-axis and K2 > 0 if a-axis orientation is energetically preferred over the c-axis. Here, \({ \langle\, {J}_{\alpha } \rangle }_{\alpha }\) is shorthand for \(\langle {\psi }_{0,\alpha }\left\vert\, {J}_{\alpha }\right\vert {\psi }_{0,\alpha } \rangle\), where \(\left\vert {\psi }_{0,\alpha }\right. \rangle\) is the ground state of the total Hamiltonian when an exchange field of magnitude Eex is applied along the α-direction.

The charge density was calculated following the method in ref. 51. Figure 4 shows iso-surfaces of the charge density at 75% of its maximum value. The anisotropic part of the charge density is exaggerated 20× in Fig. 4e according to the expression 20 ⋅ (R − R0) + R0, where R0 corresponds to 10 Bohr radii. More details are given in Supplementary Figs. 15, 16.

Spin model calculations

A model Hamiltonian, Eq. (1), is introduced from the viewpoint of symmetry in Supplementary Section I, to explain the essential experimental results. Here, magnetic frustration is lifted and the separation of spin components by modulation (q-) vectors is explained naturally by the off-diagonal \({E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ab}\) and \({E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ac}\) terms. An analytic solution is obtained in Fourier space, and variational calculations are carried out based on the spin ansatz and ignoring higher harmonics, for simplicity,

$$ \left({S}_{n}^{a},\, {S}_{n}^{b},\, {S}_{n}^{c}\right) \\ =\left({(-1)}^{n}m,\sqrt{1-{m}^{2}}\cos [(\pi+{q}_{{{{{{{{\rm{CDW}}}}}}}}}){z}_{n}],\sqrt{1-{m}^{2}}\sin [(\pi+{q}_{{{{{{{{\rm{CDW}}}}}}}}}){z}_{n}]\right),$$
(4)

In these terms, the energy is given by

$$\begin{array}{r}E[m]=-{J}_{2}^{{{{{{{{\rm{AFM}}}}}}}}}{m}^{2}-\frac{{E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ab}+{E}_{{{{{{{{\rm{CDW}}}}}}}}}^{ac}}{2}m\sqrt{1-{m}^{2}}\end{array}$$
(5)

and easily optimized at m = m* satisfying δE[m*] = 0. The optimal antiferromagnetic moment m* gives the squared intensities \({I}_{{{{{{{{\rm{AFM}}}}}}}}}={\left\vert {S}^{a}(q=\pi )\right\vert }^{2}={m}^{2}\) and \({I}_{{{{{{{{\rm{cyc}}}}}}}}}={\left\vert {S}^{b}(q=\pi+{q}_{{{{{{{{\rm{CDW}}}}}}}}})\right\vert }^{2}+{\left\vert {S}^{c}(q=\pi+{q}_{{{{{{{{\rm{CDW}}}}}}}}})\right\vert }^{2}=(1-{m}^{2})/2\), depicted in Fig. 4c.